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PATENT 

Attorney Docket No.: 16869P-094600US 
Client Ref. No.: 340101041US2 

THREE DATA CENTER ADAPTIVE REMOTE COPY 

CROSS REFERENCE TO RELATED APPLICATIONS 
[01] This application is a continuation in part of U.S. Application No. 09/854,125, filed 
5 May 10, 2001 and of U.S. Application No. 10/096,375, filed March 8, 2002 the disclosures of 
are included herein by reference for all purposes. 

BACKGROUND OF THE INVENTION 
[02] The present invention relates in general to a storage apparatus system for carrying out 

10 the copy (the remote copy) of data to another storage apparatus system which is located in an 
out-of-the-way place. More particularly, the invention relates to an information processing 
system for carrying out the remote copy which is of the type in which the order of writing 
data from a host computer to a storage apparatus system does not match the order of 
transmitting data from a storage apparatus system having the data written thereto to another 

15 storage apparatus system. 

[03] In an information processing system having a host computer and a plurality of storage 
apparatus systems, as the technology of carrying out the copy of data between the storage 
apparatus systems, there is known the remote copy. 

[04] By the remote copy is meant the technology in which a plurality of storage apparatus 
20 systems which are located physically apart from one another carry out the copy of the data, 
i.e., the double writing of the data between the storage apparatus systems without 
interposition of the host computer. 

[05] The storage apparatus system is a system including a plurality of storage apparatuses 
and a controller for controlling these storage apparatuses. 

25 [06] In the information processing system which carries out the remote copy, the storage 
apparatus systems which are respectively arranged in the places physically remote from one 
another are electrically connected to one another through dedicated lines or public lines. Of 
logical storage areas (hereinafter, referred to as "logical volumes", when applicable) which a 
certain storage apparatus system has, the logical volume having the same capacity as that of 

30 the logical volume subjected to the remote copy (hereinafter, referred to as "the source 

volume" for short, when applicable) is ensured in the storage apparatus system to which the 
source volume is copied. This ensured logical volume (hereinafter, referred to as "the 



destination volume", when applicable) is formed in such a way as to show one-to-one 
correspondence relation with the source volume. 

[07] The data of the source volume is copied to the destination volume through the 
associated one of the dedicated lines or public lines. 
5 [08] When the data contained in the source volume is updated, the updated data is 
transferred to the storage apparatus system having the destination volume through the 
associated one of the dedicated lines or the like and the updated data is also written to the 
destination volume corresponding to the source volume. 

[09] If the technique of the remote copy is employed, then in the information processing 
1 0 system having a plurality of storage apparatus systems, the logical volume of the same 
contents can be held in a plurality of storage apparatus systems. 

[10] The technique relating to the remote copy is disclosed in U.S. Patent No. 5,742,792. 
In U.S. Patent No. 5,742,792, the technique called the adaptive copy is further disclosed. 
[11] By the adaptive copy is meant one of the remote copy techniques. The adaptive copy 
1 5 is the remote copy method wherein before the data written from the host computer to the 
local storage device is copied to the remote storage device, the information exhibiting the 
completion of write is returned back to the host computer. 

[12] In the adaptive copy, the transmission order of data is not serialized, and hence the 
order of writing the data to the source volume by the host computer may be different from the 

20 order of transferring these data to the destination volume in some cases (hereinafter, such 
remote copy is referred to as the remote copy of "no guarantee to order", when applicable). 
[13] When the host computer writes repeatedly data to the same location in the destination 
volume on the basis of that property, only the data which has been written thereto lastly can 
be transmitted to the storage apparatus system having the destination volume. Therefore, the 

25 load on the network such as the dedicated line between the storage apparatus systems can be 
reduced. 

[14] On the other hand, when the host computer in which the file system used in the so- 
called open system is incorporated writes the data to the storage apparatus system, in general, 
the buffer and the like provided in the host computer, whereby an instruction to transfer the 
30 data from an application program to the file system is made asynchronously with the 
operation of writing the data to the storage apparatus system, 

[15] But, in the case that the data in the file is destroyed due to various problems, in order 
to keep the coherency of the file system structure, with respect to at least the directory and the 
meta-data such as i-node which are used to manage the file system, the operation of issuing 



2 



an instruction to transfer the data, i.e., the directory and the meta-data from the host computer 
to the storage apparatus system is carried out synchronously with issuing of the write 
command from the application program running on the host computer to the file system. The 
above-mentioned technique is disclosed in an article of "The Design and Implementation of a 
5 Log-Structured File System", Mendel Resenblum and John K. Ousterhout, ACM 
Transactions on Computer Systems, Vol. 10, No. 1, February 1992, page 29. 
[16] By executing such a processing, even if the data in the file which is buffered in the 
host computer is lost due to the asynchronous writing by an abrupt power source shutdown or 
the like, the meta-data is not lost at all. So, the coherency of the file system structure is kept 
10 and the damage can be kept to a minimum even though the data itself is lost. 



SUMMARY OF THE INVENTION 
[17] The present invention comprises a storage apparatus configuration as used in a three 
data center architecture. The storage apparatus comprises first and second data centers in 

1 5 close proximity, and a remote data center. The first data center can copy an image of data 
stored therein to the remote data center without affecting access to storage in the first data 
center. The second data center can perform a takeover operation if the first data center is 
down. The second data center can ascertain when the remote data center has updated its data 
state so that the second data center can copy an image of data stored therein to the remote 

20 data center without affecting access to storage in the second data center. 



BRIEF DESCRIPTION OF THE DRAWINGS 
[18] Aspects, advantages and novel features of the present invention will become apparent 
from the following description of the invention presented in conjunction with the 
25 accompanying drawings, wherein: 

Fig. 1 is a block diagram showing a configuration of a computer system to 
which a first embodiment of the present invention is applied; 

Fig. 1 A illustrates a variation of the embodiment shown in Fig. 1 as used in a 
three data center configuration; 
30 Fig. IB shows some detail for the three data center configuration shown in 

Fig. 1A; 

Fig. 1C illustrates a failover scenario with recovery; 

Fig. ID illustrates a second failover scenario with recovery; 

Fig. IE shows a illustrative implementation of a transfer state bit map; 
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Figs. 2 A to 2C are respectively schematic views useful in explaining the flow 
of the data in the first embodiment; 



5 embodiment; 

Fig. 5 is a flow chart useful in explaining the main remote copy processing in 
the first embodiment; 

Fig. 6 is a flow chart useful in explaining the sub-remote copy processing in 
the first embodiment; 

1 0 Fig. 7 is a flow chart useful in explaining the main freezing processing in the 

first embodiment; 

Fig. 8 is a flow chart useful in explaining the sub-freezing processing in the 
first embodiment; 

Fig. 9 is a flow chart useful in explaining the sub-recovery processing in the 
1 5 first embodiment; 

Fig, 10 is a flow chart useful in explaining the reading processing in the first 

embodiment; 

Fig. 1 1 is a schematic view useful in explaining the flow of the freeze 
instruction corresponding to the first embodiment; 
20 Fig. 12 is a block diagram showing a configuration of a computer system to 

which a second embodiment of the present invention is applied; 

Figs. 13A to 13D are respectively schematic views useful in explaining the 
flow of the data in the second embodiment; 

Fig. 14 is a schematic view useful in explaining the flow of a freeze 
25 instruction in the second embodiment; 

Fig. 15 is a flow chart useful in explaining the write processing A in the 
second embodiment; 

Fig, 16 is a flow chart useful in explaining the main freezing processing A in 
the second embodiment; 

30 Fig. 17 is a flow chart useful in explaining the sub-remote copy processing A 

in the second embodiment; 

Fig. 18 is a flow chart useful in explaining the sub-freezing processing A in 
the second embodiment; 



Fig. 3 is 
Fig. 4 is 



a block diagram showing an internal configuration of a DKC; 
a flow chart useful in explaining the write processing in the first 
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Fig. 19 is a block diagram showing a configuration of a computer system to 
which a third embodiment of the present invention is applied; 

Figs. 20A to 20D are respectively schematic views useful in explaining the 
flow of the data in the third embodiment; 
5 Fig. 21 is a schematic view useful in explaining the flow of a freeze 

instruction in the third embodiment; 

Fig. 22 is a flow chart useful in explaining the main write processing B in the 
third embodiment; 

Fig. 23 is a flow chart useful in explaining the main freezing processing B in 
1 0 the third embodiment; 

Fig. 24 is a flow chart useful in explaining the main copy processing B in the 
third embodiment; 

Fig. 25 is a flow chart useful in explaining the main remote copy processing B 
in the third embodiment; 
15 Fig. 26 is a flow chart useful in explaining the main freezing completion 

processing in the third embodiment; 

Fig. 27 is a flow chart useful in explaining the sub-freezing processing in the 
third embodiment; and 

Fig. 28 is a flow chart useful in explaining the sub-remote copy processing B 
20 in the third embodiment. 



DESCRIPTION OF THE SPECIFIC EMBODIMENTS 
[19] The preferred embodiments of the present invention will hereinafter be described in 
detail with reference to the accompanying drawings. 
25 [20] Fig. 1 is a block diagram showing a configuration of a first embodiment of a computer 
system 1000 to which the present invention is applied. 

[21] The computer system 1000 includes a host computer 1010, a main storage apparatus 
system 1 180, a substorage apparatus system 1 190 for carrying out the remote copy between 
the main storage apparatus system 1 180 and the sub-storage apparatus system 1 190, a sub- 
30 host computer 1020 and a network 1050 through which the main storage apparatus system 
1 1 80 and the sub-storage apparatus system 1 190 are linked to each other. 
[22] The substorage apparatus system 1 190 is used when the main storage apparatus 
system 1 180 becomes unusable due to the calamity or the like. The sub-host computer 1020, 
when the host computer 1010 or the main storage apparatus system 1 180 becomes unusable, 
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takes over the processing which the host computer 1010 is expected essentially to execute, 
using the data in the sub-storage apparatus system 1 190. 

[23] For the network 1050, it is assumed that it is the shared network such as the Internet, 
and the transfer rate of the data is not so high. In the present embodiment, it is assumed that 
5 the remote copy between the main storage apparatus system 1 180 and the substorage 

apparatus system 1 190 is carried out with the no guarantee to order while no heavy load is 
applied to the network 1050. By the way, the data transfer rate of the network 1050 may be 
high as well. 

[24] The main storage apparatus system 1 180 includes a main disk unit 1 130 in which the 
10 data transferred from the host computer 1010 is stored, and a main disk controller 1030 

(hereinafter, referred to as "a DKC" for short, when applicable) for controlling the main disk 
unit 1130. 

[25] In the present embodiment, the host computer 1010 issues an instruction to the main 
storage apparatus system 1 180 in such a way as to carry out "a freezing operation" 

15 (hereinafter, referred to as "the freeze instruction" when applicable). 

[26] The meaning of "freezing" is to hold, for the substorage apparatus system 1 190, the 
same volume image as that of the main storage apparatus system 1 1 80 at a time point when 
the host computer 1010 has issued the instruction. Thus, when the host computer 101 issues 
a freeze instruction, a volume image of the main storage apparatus system 1 180 is produced 

20 on the substorage apparatus system 1 1 90. 

[27] The main disk unit 1 130 includes a main volume 1 120 and a main differential volume 
1 140. Each of the main volume and the main differential volume may be either the logical 
volume or the physical volume. In the case of the logical volume, as in the present 
embodiment, a plurality of volumes may be present in the main disk unit in some cases. On 

25 the other hand, in the case of the physical volume, the main disk unit further includes a 

plurality of disk units each of which constitutes the physical volume. In addition, in the case 
as well of the logical volume, the main disk unit may be constituted by a plurality of disk 
units. 

[28] In the main volume 1 120, there is stored the data which has been transferred from the 
30 host computer 1010 to the main storage apparatus system 1 180. 

[29] In the main differential volume 1 140, for a time period ranging from a time point 
when the main storage apparatus system 1 180 has received the freeze instruction from the 
host computer 1010 up to a time point when the processing of the freezing has been 
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completed (hereinafter, such a time period is referred to as "in the freezing" for short, when 
applicable), there is stored the data which has been transferred from the host computer 1010. 
[30] The main DKC 1030 has a main freezing mode 1060, a main bitmap 1070 and a main 
differential bitmap 1080 on the memory. 
5 [31] The main freezing mode 1060 shows whether or not the main DKC 1030 itself is 
executing the process of the freezing. For example, if it is determined that the main DKC 
1030 itself is executing the processing in the freezing, then the mode becomes 1. 
[32] The main bitmap 1070 and the main differential bitmap 1080 have the respective bits 
each corresponding to a block which the main volume 1 120 has. 

10 [33] The main bitmap 1070 shows the blocks in the main volume 1 120. These blocks are 
the data, which are not yet transferred to the sub-DKC 1040, among the data stored in the 
main volume 1 120 before the main storage apparatus system 1 180 receives the freeze 
instruction from the host computer 1010. In other words, main bitmap 1070 identifies the 
blocks in main volume 1 120 which have not been copied to sub-DKC 1040 at the time a 

1 5 freeze instruction is received. 

[34] The main differential bitmap 1080 shows the blocks which contain the data which the 
main disk unit 1 130 has received from the host computer 1010 while the main DKC 1030 is 
executing the process of the freezing. 

[35] The data is stored in the log-structured postscript type file format in the main 
20 difference volume 1 140. The block numbers exhibiting corresponding blocks in the main 
volume 1 120 are also contained in the stored data. 

[36] In the case where the write data to be written to the main volume 1 120 is received 
while the main DKC 1030 is executing the process of the freezing, the main DKC 1030 
checks the main differential bitmap 1080. When the write data to be written has already been 

25 present in the main differential volume 1 140, the data which was written to the main 
differential volume 1 140 prior thereto is cancelled in such a way that a plurality of data 
written to the same block are not present in the main differential volume 1 140. 
[37] The substorage apparatus system 1 190 includes a sub-disk unit 1 160 in which the data 
which has been remote-copied from the main storage apparatus system 1 180 is stored, and a 

30 sub-DKC 1040 for controlling the sub-disk unit 1 160. 

[38] The sub-disk unit 1 160 includes a sub-volume 1 150, a sub-differential volume A 1 170 
and a subdifferential volume B 1 175. 

[39] In the sub-volume 1 150, there is stored the volume image of the main volume 1 120 at 
a certain time point, more specifically, at a time point when the host computer 1010 
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previously issued the freeze instruction to the main storage apparatus system 1 1 80. The 
blocks of the sub-volume 1 150 and the main volume 1 120 show one-to-one correspondence. 
[40] In the sub-differential volume A 1 170, there is stored the data which was written to 
the main storage apparatus system 1 180 on and after the host computer 1010 previously 
5 issued the freeze instruction to the main storage apparatus system 1 180. 

[41] The sub-DKC 1040 includes a sub-freezing mode 1090, a sub-bitmap 1 100 and a sub- 
differential bitmap 1 1 10 on a memory (not shown). 

[42] The value of the sub-freezing mode 1090 becomes zero when the sub-DKC 1040 does 
not execute the process of the freezing, and becomes 1 or 2 when the sub-DKC 1040 is 
10 executing the process of the freezing. 

[43] The sub-bitmap 1 100 and the sub-differential bitmap 1 1 10 are constituted with the 
same number of bits as the number of blocks which the sub-volume 1 150 has, and 1 bit 
corresponds to 1 block. 

[44] The sub-bitmap 1 100 shows the presence or absence of the data which is stored in the 
1 5 subdifferential volume A 1 1 70. 

[45] The sub-differential bitmap 1110 shows the presence or absence of the data which is 
stored in the sub-differential volume B 1 175. 

[46] The data is respectively stored in the log-structured type file format in the sub- 
differential volume A 1 170 and the sub-differential volume B 1 175 similarly to the main 

20 differential volume 1 140. The block numbers exhibiting the corresponding blocks in the sub- 
volume 1 150 are also contained in the data which is respectively stored therein. 
[47] The sub-DKC 1040 executes the same processing as that in the main DKC 1030 to 
cancel the old data in such a way that two or more data corresponding to the same block is 
not present in the sub-differential volume A 1 170 and the sub-differential volume B 1 175. 

25 [48] While in the present embodiment, the host computer 1010 issues the freeze 

instruction, alternatively, the main DKC 1030 or the sub DKC 1040 may issue the freeze 
instruction. 

[49] The application program which runs on the host computer 1010 issues an instruction 
to write data to the main storage apparatus system 1 180. But, in actuality, there may be the 
30 case where the data is written to the cache or buffer memory of the host computer 1010 and 
hence the contents of the data stored in the main storage apparatus system 1180 become the 
state which the application program does not intend (i.e., the data which ought to have been 
written is not yet written in actual). 
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[50] If under this state, the host computer 1010 has gone down due to a shutdown of the 
power source, and the file system recovery program such as fsck is executed for the main 
storage apparatus system 1 1 80, then there is the possibility that the data which is not reflected 
in the main storage apparatus system 1 180 may be lost. 
5 [51] In order to prevent the data which is not reflected in the main storage apparatus 
system 1 1 80 from being lost, the main DKC 1030 or the like should not issue freely the 
instruction for the freezing, but the instruction for the freezing should be issued after the host 
computer 1010 has written all of the unreflected data on the cache or buffer memory to the 
main storage apparatus system 1180. 

10 [52] Figs. 2 A to 2C show the flow of the data in the present embodiment. 

[53] Fig. 2A is a schematic view showing the flow of the data in the state in which the host 
computer 1010 has not yet issued the freeze instruction, i.e., in the normal state. 
[54] The data which has been sent from the host computer 1010 is written to the main 
volume 1 120 (indicated by an arrow A 100). The data which has been newly written to the 

15 main volume 1 120 is read out from the main volume 1 120 to the main DKC 1030 to be 
transferred to the sub-DKC 1 040 to be written to the sub-differential volume A 1 1 70 
(indicated by an arrow B 110). 

[55] At an arbitrary time point in Fig. 2A, in the sub-volume 1 150, there is stored the same 
volume image as that of the main volume 1 120 at a time point when the host computer 1010 

20 issued the freeze instruction last time. 

[56] Fig. 2B is a schematic view showing the flow of the data from a time point after the 
host computer 1010 issues the freeze instruction up to a time point when the main DKC 1030 
issues a notification command reporting that all of the data has been sent (hereinafter, 
referred to as "all freezing data transmission completion", when applicable) to the sub-DKC 

25 1040. 

[57] If the data sent from the host computer 1010 is to be stored in a block of the main disk 
unit 1 130 that was already sent to the sub-DKC 1040 at the time when the host computer 
1010 issued the freeze instruction, then the data is written to the main volume 1 120 as 
indicated by arrow CI 20. On the other hand, if the data sent from the host computer 1010 is 
30 to be stored in a block which is holding data that has not yet been sent to the sub-DKC 1040, 
then that data is written to the main differential volume 1 140, as indicated by arrow D130. 
[58] Data in a block in the main volume 1 120 which has not yet been sent to the sub-DKC 
1040 at the time the freeze instruction was issued is read out from the main volume 1 120 to 
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the main DKC 1030. The data is then transferred to the sub-DKC 1040 to be written to the 
sub-differential volume A 1 170, as indicated by arrow El 40. 

[59] The main DKC 1030, in accordance with the contents of the main bitmap 1070, 
transfers all of the difference data between the main volume 1 120 and the sub-volume 1 150 
5 at a time point when the freeze instruction issued from the host computer 1010 to the sub- 
differential volume A 1 170. After completion of all of the transfers, the main DKC 1030 
informs the sub-DKC 1040 that all of the freezing data has been transferred. 
[60] At an arbitrary time point in Fig. 2B, in the sub-volume 1 150, there is stored the 
volume image having the same contents as those of the volume image which the main 
10 volume 1 120 held at a time point when the host computer 1010 issued the instruction for the 
freezing last time. 

[61] Fig. 2C is a schematic view showing the flow of the data from a time subsequent to 
when the main DKC 1030 informs the sub-DKC 1040 that all of the freezing data has been 
transferred. As will be explained, the main DKC 1030 updates the data stored in the main 
15 volume 1 120 using the data stored in the main differential volume 1 140. Also, the sub-DKC 
1040, using the data stored in the sub-differential volume A 1 170, updates the data which is 
stored in the sub-volume 1 150. Data from the host computer 1010 is now written to the main 
volume 1 120, as indicated by an arrow F150. 

[62] Data stored in the main volume 1 120 is updated based on data which is stored in the 
20 main differential volume 1 140, as indicated by an arrow G160. But, in the case where the 

block containing the data becoming an object of the update is already updated the data which 

has been transferred from the host computer 1010, the update of the data in the main volume 

1 120 corresponding to the data stored in the main differential volume 1 140 is not carried out. 

Thus, any blocks written by the host computer 1010 after all the freezing data has been 
25 transferred to sub-DKC 1040 will not be updated from the main differential volume 1 140, 

because the data from the host computer 1010 is more recent that the corresponding data in 

main differential volume 1140. 

[63] The data which has been transferred from the host computer 1010 to the main disk 
unit 1 130 after the freeze instruction from the host computer 1010 is read out from the main 
30 volume 1 120 to the main DKC 1030 to be sent to the sub-DKC 1040 to be stored in the sub- 
differential volume B 1 175, as indicated by an arrow HI 70. Thus, copying of data to the sub- 
storage apparatus system 1 1 90 resumes, this time with sub-differential volume B 1 1 75 being 
the receiving volume, instead of sub-differential volume 1 1 70. 
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[64] The sub-DKC 1040 reads out the data in the sub-differential volume A 1 170 to the 
sub-DKC 1040 to store that data in the sub- volume 1 150, as indicated by an arrow 1180. At 
an arbitrary time point in Fig. 2C, the data stored in the sub-volume 1 150 is combined with 
the data stored in the sub-differential volume A 1 170, whereby the volume image of the main 
5 volume 1 120 at a time point when the host computer 1010 issued the freeze instruction to the 
main DKC 1030 that time is reproduced. In other words, sub-volume 1 150 contains a 
volume image representing the data state of main volume 1 120 at the time the freeze 
instruction was issued. 

[65] From the foregoing, in Figs. 2A and 2B, the data of the volume image having the 
10 same contents as those of the data of the volume image of the main disk unit 1 120 at a time 
point when the freeze instruction was issued by the host computer 1010 last time is present in 
the sub-volume 1 150. In Fig. 2C, the data of the volume image having the same contents as 
those of the data of the volume image in the main disk unit 1 120 at a time point when the 
freeze instruction was issued at that time can be reproduced by combining the data stored in 
15 the sub-volume 1 150 with the data in the sub-differential volume A 1 170. 

[66] In other words, in any case, the volume images which are coherent with the volume 
images of the main disk unit 1 120 at a time point when the host computer 1010 issued the 
freeze instruction is prepared in the sub-disk unit 1 160. 

[67] Fig. 1 A shows an aspect of the present invention as incorporated in a three data center 
20 data storage system 100. This configuration is more fully discussed in U.S. Application No. 
10/096,375, and so is described briefly here. A storage sub-system is located in each of three 
data centers 102 (data center 1), 104 (data center 2), 112 (data center 3). 
[68] Data center 1 and data center 2 are located sufficiently close that synchronous data 
replication between the two data centers is practical without adverse affects as to 
25 performance. Thus, for example, data center 2 can serve as a primary data center 104 with 
data center 1 being a local remote copy data center 102. A write operation from a user 
system sent to the primary data center can be mirrored in the remote data copy center 
synchronously; i.e., the primary data center waits for an acknowledgement from the remote 
copy data center before acknowledging the user system. Synchronous operation guarantees 
30 data consistency between the primary data center and the remote copy data center. 

[69] Data center 3 is typically located a long distance away from either the primary data 
center 104 and the local remote copy data center 102. Data center 3 therefore serves as a 
"remote" data center (remote site) 1 12, that is protected against (or at least less susceptible to) 
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catastrophic events that might take out both the primary data center 104 and the local remote 
copy data center 102. 

[70] The primary data center 104 and the remote site 1 12 therefore are likely to be located 
sufficiently far apart that the data transmission delay between the two data centers 102, 1 12 
5 would render synchronous copy operations unacceptable for operational requirements (e.g., 
response time) of the primary data center, as determined for example by a system 
administrator. In accordance with this particular embodiment, the asynchronous technique of 
the present invention is employed as the copy method between the primary data center 104 
and the remote site 112. It will also be appreciated that the local remote copy data center 102 

1 0 is also configure for the asynchronous copying of the present invention. 

[71] Thus, referring to Figs. 1 and IB, the primary data center 104 can be configured with 
host computer 1010-2 and a main storage apparatus 1 180-2 comprising a main disk unit 
1 130-2 and a DKC 1060-2. Similarly, the local remote data center 102 can be configured 
with a host computer 1010-1 and a main storage apparatus 1 180-1 comprising a main disk 

15 unit 1 130-1 and a DKC 1060-1. It can be appreciated from Fig. 1 that each DKC 1030-1 and 
1030-2 comprises a main freezing mode 1060 (Fig. 1), a bit map 1070, and a differential bit 
map. The main disk unit 1130-2 comprises main volume 1 120-2 and main differential 
volume 1 140-2. Similarly, the main disk unit 1 130-1 comprises main volume 1 120-1 and 
main differential volume 1 140-1. 

20 [72] The remote site 1 12 can be configured with a sub-host computer 1020-3 and a sub- 
storage apparatus 1 190-3 comprising a sub disk unit 1 160-3 and a sub-DKC 1040-3. The 
DKC 1040-3 comprises a sub-freezing mode 1090 (Fig. 1), a sub-bit map 1 100, and a sub- 
differential bit map 1 110. The sub disk unit 1 160-3 comprises sub-volume 1 150-3, a first 
sub-differential volume 1 170-3 and a second sub-differential volume 1 175-3. 

25 [73] Fig. IB also shows that DKC 1030-2 in the primary data center 104 is configured with 
three ports 22, 24, and 26. DKC 1030-1 in the local remote copy data center 102 is 
configured with two ports 12, 14. DKC 1040-3 of the remote site 1 12 is configured with 
ports 32 and 34. Host computer 1010-2 of the primary data center 104 communicates with its 
DKC 1030-2 via port 24. Host computer 1010-1 of the local remote copy data center 102 

30 communicates with its DKC 1030-1 via port 14. Sub-host computer 1020-3 of the remote 
site 112 communicates with its DKC 1040-3 via port 34. 

[74] A host communication path 122 is provided between host computer 1010-2 of 
primary data center 104 and host computer 1010-1 of local remote copy data center 102. A 
host communication path 126 is provided between host computers 1010-1 and 1020-3. A 
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data communication path 132 is provided between DKC 1030-2 and 1030-1, via ports 22 and 
12, respectively. Similarly, a host communication path 124 is provided between host 
computer 1010-2 of primary data center 104 and sub-host computer 1020-3 of the remote 
site 1 12. A data communication path 134 is provided between DKC 1030-2 and DKC 1040- 
5 3, via ports 26 and 32 respectively. It can be appreciated that each of the communication 
paths 122, 124, 126, 132, and 134 can include a path through one or more communication 
networks, or can be a dedicated link. For example, the data communication path 132 can be a 
dedicated line for very high speed communication, while the host communication path 122 
can be via a communication network. The communication paths 124, 134 are likely to span 

10 one or more communication networks, since the remote site 1 12 is typically far away from 
either the primary data center 104 and the local remote copy data center 102. 
[75] Referring to Figs. 1 A and IB, in accordance with normal operations, the primary data 
center receives storage access requests from user systems (not shown). Data access is made 
with respect to the main storage 1 130-2 of the primary data center 104. Data associated with 

15 a write operation is mirrored on the local remote copy data center 102 in a synchronous 

manner over data communication path 132. The data is also mirrored at the remote site 1 12, 
but in an asynchronous manner as discussed above in connection with Figs. 1 and 2A - 2C, 
using data communication path 134. 

[76] Occasionally, the primary data center 104 becomes unavailable. For example, routine 
20 maintenance activity might require that the primary data center be taken offline for a period 
of time. When some activity occurs that causes the primary data center 104 to go offline, a 
failover (takeover) operation can be performed so that the local remote copy data center 102 
can take over the functions of the primary data center, such as receiving and processing 
storage access requests. For convenience, the local remote copy data center 102 can now be 
25 referred to as the "takeover center." 

[77] Another function that is taken over by the takeover center 102 is data mirroring to the 
remote site 112. From the discussion of asynchronous copy given above, it can be 
appreciated that the data state of the data at the primary data center 104 (and also the takeover 
center 102) is different from the data state of the remote site 112. Referring to Fig. IB, the 
30 data state of the data on the sub-volume 1 150-3 of the remote site 1 12 is different from the 
data state of the main volume 1 120-2 of the primary data center. 

[78] Consider the case that host computer 1010-2 in the primary data center 104 is down, 
and that the storage apparatus 1 180-2 remains operational. In this case, the storage apparatus 
1 180-2 can still communicate with the remote site 1 12, as indicated in Fig. 1A. In this 
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embodiment of the invention, the takeover center 102 can be configured to communicate a 
freeze operation to the DKC 1030-2 of the primary data center 104 (via the communication 
path 132, for example). Thus, even if the host unit 1010-2 is not operational, the image 
contained in its main volume 1 120-2 can be copied to the sub-storage apparatus 1 190-3 
5 according to the processing of the freeze command discussed above. Eventually, the remote 
site 1 12 will have an image of the primary data center 104 (and therefore, of the takeover 
center 102) at the time of failover. 

[79] Since the exchange of data during freeze operation processing occurs only between 
storage apparatus 1 180-2 of the primary data center 104 and the sub-storage apparatus 1 190-3 

10 of the remote site 1 12, the takeover center 102 has no information as to the progress of the 
exchange. Therefore, an aspect of the present invention, as embodied in this particular 
embodiment, Fig. 1 A shows a data transfer state inquiry that is made by the takeover center 
102 to the remote site 112 over communication path 126. As discussed above in connection 
with Fig. 2B, after completion of all the transfer of data, DKC 1030 (in the primary data 

15 center 104) communicates a completion signal to DKC 1040 (in the remote site 112) 
indicating that the freezing data (i.e., image data) has been transferred. Thus, prior to 
receiving the completion signal, the remote site 112 can respond to a data transfer state 
inquiry (from the takeover center) with an indication that the data transfer has not completed. 
After receiving the completion signal, the remote site can respond to a data transfer state 

20 inquiry (from the takeover center) with an indication that the data transfer has completed. 

[80] Referring to Figs. 1 A, IB, and IE, in another aspect of the invention, the data transfer 
state inquiry can employ a transfer state/bit map mechanism. This mechanism is discussed 
more fully in U.S. Application No. 10/096,375. Consequently, only a brief discussion will be 
presented herein. Fig. IE is a diagram showing an example table for the transfer state/bit 

25 map (hereinafter referred to as a bit map, as needed). A bit map is provided in each of the 

data centers 102, 104, 112. Thus, the takeover center 102 includes a transfer state/bit map 1 1 
and the remote site 1 12 includes a transfer state/bit map 3 1 . The bit map is used between any 
two data centers that will be paired at failover during recovery from a disaster or a blockage. 
For example, in Fig. 1 A, the takeover center 102 and remote site 1 12 are paired to cope with 

30 the unavailability of the primary data center 1 04. 

[81] The transfer state/bit map is an embodiment of an aspect of the invention to facilitate 
the paired data centers. Each bit map is employed to manage a difference with the logical 
volume of its paired partner. A block number field 182 corresponds to a block that is a 
minimum data unit for managing the update of the logical volume. 



[82] The host I/O data units need not be the same size as the data unit of a block. The unit 
of the host I/O is normally 512 bytes, at the minimum, and an upper limit is also set; 
however, these are variable. The bit map is slightly smaller than 50 KB or around 700 kB; 
however, it can have various sizes ranging from 20 KB to 1000 KB. One bit map does not 
5 always correspond to one block of host I/O data. 

[83] When the contents of the block corresponding to the block number are updated, 
differential management is conducted for all the data for the pertinent block number, and at 
the time of synchronization (resync), all the data for the block number is transmitted. For 
each block number, the bit map is used as the unit for which the logical volume is updated. 

10 An "Update" information to be transmitted to another logical volume is waited for, so that 

only the updated block need be transmitted in order to reconstruct (re-synchronize) the pair of 
logical volumes used for remote copy. In other words, when the Update flag 184 is ON (1 in 
the embodiment in Fig. IE), it means that the pertinent data is the transmission target. Once a 
normal Update is performed by the command unit of the host, the Update flag is set to 0, 

1 5 based on the counter value of 0. 

[84] The bit map has a counter value field 1 86 whereat updates repeated multiple times are 
recorded using the same block number. For example, the counter value is 0 for no update, or 
is 3 when the updating was repeated three times. When the size of a data block represented 
by a block number is larger than a data block updated by the host, the counter value is 

20 employed so that only the updated data can be transmitted to the logical volume partner. 
[85] A block number and a counter value specified in "data transfer state inquiry 
command" are compared with the block number and the counter values of the bit map of the 
data center that receives the transfer state inquiry. If the counter value stored in the data 
center is equal to or greater than the counter value included in the transfer state inquiry", then 

25 the value is transmitted to the requesting data center and the counter value of the bit map of 
the requesting data center is decremented by one. If the counter value in the receiving data 
center is smaller than the counter value in the transfer state inquiry, then the counter value of 
the bit map in the requesting data center is unchanged. 

[86] If the counter value of the bit map of the receiving data center is equal to or greater 
30 than the counter value in the received transfer state inquiry, then the data updating status 
indicates that the data has already been stored. If the counter value of the bit map in the 
receiving data center is less than the counter value indicated in the transfer state inquiry, then 
it means that data has not yet been received. 
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[871 The counter value in Fig. IE is finite, and when, for example, one byte is allocated as 
the counter value, the management process cannot be performed more than 256 times. In this 
example, when the same block has been updated over 256 times, the counter value is not 
incremented any longer, and the Update flag is set permanently. That is, in Fig. IE, 
5 information representing "Over Flow" is stored in the counter value. When this overflow 
condition arises, a release (i.e., entering a value of 0) of the Update flag of the block, which is 
specified in the bit map and which is permanently set, is not performed until the data center 
having this bit map acknowledges that the data transmission to the partner data center has 
completed, namely, the data has been received and is stored. 

10 [88] Thus, based on the transfer state bit maps, the takeover center 102 can determine that 
the data state of the remote site 1 12 is updated and is the same as the data state of the 
takeover center. At this point, the takeover center 102 can establish a data communication 
path 136 with the remote site 112. Further details of this aspect of the invention are discussed 
in U.S. Application No. 10/096,375. The remote site 1 12 now has the same image as the 

1 5 primary data center, and hence the takeover center 102. Since the takeover center is 

configured similarly to the primary data center 104, asynchronous copying from the takeover 
center 102 to the remote site 1 12 can be performed in accordance with the present invention 
using the communication path 136. 

[89] Referring now to Fig. 1C, consider the case where both the host unit 1010-2 and 
20 storage apparatus 1 1 80-2 become unavailable. The remote site 1 12 would not be able to 
update its image of the primary data center 104 because storage apparatus 1 180-2 is 
unavailable. In accordance with another aspect of the present invention, when the local 
remote copy data center 102 becomes the takeover center, it attempts to communicate a 
freeze operation to the storage apparatus 1 180-2. Failing to do so (because it is unavailable), 
25 the takeover center 102 establishes the communication path 136 to the remote site 112. Since 
the data state of the takeover center is the same as that of the primary data center 104, the 
takeover center can initiate a freeze operation with the remote site 112, using its DKC 1030-1 
and its main volume 1 120-1 to update the image at the remote site. After that, asynchronous 
copying from the takeover center 102 to the remote site 1 12 can be performed in accordance 
30 with the present invention using the communication path 136. 

[90] Referring now to Fig. ID, consider again the case where both the host unit 1010-2 and 
storage apparatus 1 1 80-2 become unavailable. The remote site 112 would not be able to 
update its image of the primary data center 104 because storage apparatus 1 180-2 is 
unavailable. In another aspect of the present invention, as illustrated in the embodiment 
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shown in Fig. ID, data journaling is provided. During normal operation, the primary data 
center 104 communicates journal data 192 to the local remote copy data center 102, in 
addition to the data that is mirrored at the local remote copy data center. Each time the 
primary data center issues a freeze operation, it will reset its journal data. The journal data, 
5 likewise, is reset at the local remote copy data center 102. 

[91] When the primary data center becomes unavailable, the locate remote copy data 
center 102 becomes the takeover center and attempts to initiate a freeze operation with the 
storage apparatus 1 180-2 of the downed primary data center. Failing to do so, the takeover 
center 102 established the communication path 136 to the remote site 1 12. The takeover 

10 center then communicates its journal data 194 to the remote site 1 12; for example, over the 
communication path 136. This allows the remote site 1 12 to update its image to be the same 
as the image in the takeover center 102. The transfer state/bit map mechanism described in 
U.S. Application NO. 10/096,375 can be used to allow the takeover center to determine when 
the remote site 1 12 has completed its update operations with the journal entries. 

15 [92] Fig. 3 is a block diagram of the main DKC 1030. 

[93] The main DKC 1030 includes a host interface 1210, a drive interface 1230, a network 
interface 1220, a CPU 1240, a ROM 1250, a RAM 1260, and a direct memory access 
controller 1270. 

[94] The program which runs on the CPU 1240 is stored in the ROM 1250. 
20 [95] The main freezing mode 1060, the main bitmap 1070, and the main differential 

bitmap 1080 are stored in the RAM 1260. The RAM is also used as the cache memory. 

[96] The CPU 1240 controls the main storage apparatus 1 180. In the CPU 1240, a 

multitask operating system is running, so a write processing 2000, a main freezing processing 

2600 and the like can be processed in parallel with one another. 
25 [97] The sub-DKC 1040 has also the same configuration as that of the main DKC 1030. 

But, in the sub-DKC 1040, the host interface 1210 is electrically connected to the sub-host 

computer 1020, and the drive interface 1230 is electrically connected to the sub-disk 1160. 

In the RAM 1260, the sub-freezing mode 1090, the sub-bitmap 1 100 and the sub-differential 

bitmap 1 1 10 are stored. 

30 [98] Fig. 4 is a flow chart of the write processing 2000 which is executed in the main DKC 
1030 when the host computer 1010 sends the write command to write data and the data to be 
written (hereinafter, referred to as "the write data" for short, when applicable) to the main 
storage apparatus system 1 180. 
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[99] The main DKC 1030 receives the write data through the host interface 1210 (Step 
2010) to judge whether or not the main freezing mode 1060 stored in the RAM 1260 is in the 
ON state (Step 2020). 

[100] If it is determined in Step 2020 that the main freezing mode 1060 is in the OFF state, 
5 then the main DKC 1030 sets to 1 the bit of the main bitmap 1070 corresponding to the block 
having the main volume 1 120 to which the write data received by the main DKC 1030 is 
written (Step 2030). 

[101] The main DKC 1030 controls the main disk unit 1 130 in such a way that the write 
data is written to the main volume 1 120 (Step 2040). The main DKC 1030 itself judges 

10 whether or not a main remote copy processing 2200 is being executed (Step 2050). If it is 
determined in Step 2050 that the main remote copy processing 2200 is not being executed, 
then the remote processing is completed after the main DKC 1030 has executed the main 
remote copy processing 2200. On the other hand, if it is determined in Step 2050 that the 
main remote copy processing 2200 is being executed, then the write processing 2000 is 

15 completed. 

[102] If it is determined in Step 2020 that the main freezing mode 1060 is in the ON state, 
then the main DKC 1030 judges whether or not the bit of the main bitmap 1070 
corresponding to the block to which the received write data is to be written is 1 (Step 2060). 
If it is determined in Step 2060 that the corresponding bit is zero, then the processing in Step 

20 2030 is executed. On the other hand, if it is determined in Step 2060 that the corresponding 
bit is 1, then the bit of the main differential bitmap 1080 corresponding to the block in the 
main volume 1 120 in which the received write data is to be stored is set to 1 (Step 2070). 
[103] The main DKC 1030 controls the main disk unit 1 130 in such a way that the block 
number information in the main volume 1 120 is added to the received write data, and then the 

25 write data is written to the main differential volume 1 140 (Step 2080). Thereafter, the 
processing in Step 2050 is executed. 

[104] Fig. 5 is a flow chart in explaining a main remote copy processing 2200 which the 
main DKC 1030 executes. 

[105] The main remote copy processing 2200 is the processing which is called from the 
30 above-mentioned write processing 2000 and from the main freezing processing 2600 which 
will be described later. 

[106] The main DKC 1030 judges whether or not the corresponding bit of the main bitmap 
1070 is 1 and also whether or not the block which contains the data stored in the RAM 1260 
is present in the main volume 1 120 (Step 2230). If it is determined in Step 2230 that the 
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block corresponding to the condition is present therein, then the main DKC 1030 specifies the 
block of interest to execute the processing in Step 2260. 

[107] On the other hand, if it is determined in Step 2230 that the block corresponding to the 
condition is absent, then the main DKC 1030 judges whether or not the bit of 1 is present in 
5 the main bitmap 1070 (Step 2240). If it is determined in Step 2240 that the bit of 1 is not 
present in the main bitmap 1070, then the main remote copy processing 2200 is completed. 
[108] On the other hand, if it is determined in Step 2240 that the bit of 1 is present in the 
main bitmap 1070, then the main DKC 1030 specifies the block corresponding to the bit of 1 
in the main bitmap 1070 to read out the data in the block thus specified from the main volume 

10 1 120 to store the data thus read out in the RAM 1260 (Step 2250). 

[1 09] The main DKC 1 030 makes zero the bit of the main bitmap 1 070 corresponding to the 
specified block (Step 2260). The main DKC 1030 transfers the data, which has been read 
out, to the sub-DKC 1040 through the network interface 1220 (Step 2270). 
[110] After having received the report, from the sub-DKC 1040, that the sub-DKC 1040 

15 received the data of interest (Step 2280), the main DKC 1030 returns back to the processing 
in Step 2230. 

[Ill] Fig. 6 is a flow chart useful in explaining a sub-remote copy processing 2400 which 
the sub-DKC 1040 executes at the time when the main DKC 1030 has sent the data to the 
sub-DKC 1040. 

20 [112] The sub-DKC 1 040 receives the data which has been sent from the main DKC 1 030 
through the network interface 1220 (Step 2410), and then transmits the report exhibiting the 
reception of the data to the main DKC 1030 (Step 2420). 

[113] The sub-DKC 1040 judges whether or not the value of the sub-freezing mode 1090 
stored in the RAM 1260 is 2 (Step 2430), and if it is determined in Step 2430 that the value 
25 of interest is not 2, controls the sub-disk unit 1 160 in such a way that the received data is 
written to the sub-differential value A 1 170 (Step 2440). 

[114] The sub-DKC 1040 makes 1 the bit of the sub-bitmap 1 100 corresponding to the 
block having the written data to complete the sub-remote processing 2400 (Step 2450). 
[115] On the other hand, if it is determined in Step 2430 that the value of the sub-freezing 
30 mode 1090 is 2, then the sub-DKC 1040 controls the sub-disk unit 1 160 in such a way that 
the received data is written to the sub-differential value B 1 175 (Step 2460). 
[116] The sub-DKC 1040 sets to 1 the bit of the sub-differential bitmap 1110 corresponding 
to the block having the written data to complete the sub-remote copy processing 2400 (Step 
2470). 
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[117] Fig. 7 is a flow chart useful in explaining a main freezing processing 2600 which the 
main DKC 1030 executes at the time when the host computer has issued the freeze 
instruction. 

[118] After having received the freeze instruction from the host computer 1010 through the 
5 host interface 1210, the main DKC 1030 turns ON the main freezing mode 1060 which is 
stored in the RAM 1260 (Step 2610). 

[119] The main DKC 1030 transmits the freeze instruction to the sub-DKC 1040 (Step 
2620). The main DKC 1030 judges whether or not the main DKC 1030 itself is executing the 
remote copy processing 2200 (Step 2625). 

10 [120] If it is determined in Step 2625 that the main DKC 1030 itself is executing the remote 
copy processing 2200, then the main DKC 1030, after having carried out the step of waiting 
for some time (e.g., several milliseconds) (Step 2635), executes the processing in Step 2640. 
[121] On the other hand, if it is determined in Step 2625 that the main DKC 1030 itself is 
not executing the remote copy processing 2200, then the main remote copy processing 2200 

1 5 is executed (Step 2630) to execute the processing in Step 2640. 

[122] In Step 2640, the main DKC 1030 judges whether or not all of the bits of the main 
bitmap 1070 are zero. If the bit exhibiting 1 is present in the main bitmap 1070, then since 
the data to be sent to the sub-DKC 1040 still remains in the main disk unit 1 130, the main 
DKC 1030 executes again the processings after Step 2625 until all of the bits of the main 

20 bitmap 1070 have become zero. 

[123] On the other hand, if it is determined in Step 2640 that all of the bits of the main 
bitmap 1070 are zero, then the main DKC 1030 transmits the information, that the freezing 
data has already been transmitted, to the sub-DKC 1040 (Step 2650). 
[124] The main DKC 1030 judges whether or not all of the values of the main differential 

25 bitmap 1080 are zero (Step 2660). 

[125] If it is determined in Step 2660 that all of the values of the main differential bitmap 
1080 are not zero, then the main DKC 1030 controls the main disk unit 1 130 in such a way 
that the data of the block, in which the bit in the main differential bitmap 1080 is 1, is read 
out from the main differential volume 1 140 (Step 2670) to be written to the block to which 

30 the main volume 1 120 corresponds (Step 2680). 

[126] The main DKC 1030 sets to zero the bit of the main differential bitmap 1080 
corresponding to the block containing the written data (Step 2690), while it sets to 1 the 
corresponding bit of the main bitmap 1070 to return back to the processing in Step 2660 (Step 
2695). 
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[127] On the other hand, if it is determined in Step 2660 that all of the values of the main 
differential bitmap 1080 are zero, then the main DKC 1030 waits for the completion report 
from the sub-DKC 1040 (Step 2700). After having received the completion report 
transmitted from the sub-DKC 1040 (Step 2710), the main DKC 1030 turns OFF the main 
5 freezing mode 1060 which is stored in the RAM 1260 (Step 2720) to transmit a signal 

exhibiting the freezing processing completion report to the host computer 1010 to complete 
the main freezing processing 2600 (Step 2730). 

[128] Upon completion of the processing in Step 2650, the processing in Step 2700 may be 
executed, and the update processing for the main differential volume (Steps 2660 to 2695) 
1 0 may be executed after completion of the main freezing processing. 

[129] Fig. 8 is a flow chart useful in explaining a sub-freezing processing 2800 which the 
sub-DKC 1040 executes at the time when the main DKC 1030 has issued the freeze 
instruction to the sub-DKC 1040. 

[130] After having received the freeze instruction command from the main DKC 1030, the 
15 sub-DKC 1040 sets to 1 the sub-freezing mode 1090 which is stored in the RAM 1260 (Step 
2810) to wait for the report, exhibiting that all of the freezing data has already been 
transmitted, from the main DKC 1030 (Step 2820). 

[131] After having received the report exhibiting the completion of the transmission of all 
of the freezing data from the main DKC 1030, the sub-DKC 1040 sets to 2 the sub-freezing 
20 mode 1090 (Step 2825). 

[132] The sub-DKC 1040 judges whether or not the corresponding bit of the sub-bitmap 
1 100 stored in the RAM 1260 is 1 and also whether or not the corresponding block having 
the data is present on the RAM 1260 (Step 2830). 

[133] If it is determined in Step 2830 that the corresponding block is present on the RAM 
25 1260, then the sub-DKC 1040 executes the processing in Step 2860. On the other hand, if it 
is determined in Step 2830 that the corresponding block is not present on the RAM 1260, 
then it is determined whether or not the block in which the corresponding bit of the sub- 
bitmap 1 100 is 1 is present (Step 2840). 

[134] If it is determined in Step 2840 that the block in which the corresponding bit of the 
30 sub-bitmap 1 100 is 1 is present, then the sub-DKC 1040 reads out the data corresponding to 
the bit as 1 from the sub-differential volume A 1 170 to the RAM 1260 (Step 2850) to write 
the data of interest to the block of the sub-volume 1 150 corresponding to the data read to the 
RAM 1260 (Step 2860). 
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[135] The sub-DKC 1040 sets to zero the bit of the sub-bitmap 1 100 corresponding to the 
data which has already been written (Step 2870) to return back to the processing in Step 
2830. 

[136] On the other hand, if it is determined in Step 2840 that the block in which the 
5 corresponding bit of the sub-bitmap 1 100 is 1 is absent, then the sub-DKC 1040 sets to zero 
the sub-freezing mode 1090 (Step 2880) to transmit the report exhibiting the completion of 
the sub-freezing processing to the main DKC 1030 (Step 2890) to complete the sub-freezing 
processing 2800. 

[137] Fig. 9 is a flow chart useful in explaining a sub-recovery processing 3000 which the 
10 sub-DKC 1040 executes in accordance with the instruction from the sub-host computer 1020 
in the case where the host computer 1010 and the main storage apparatus 1 180 become both 
unusable at all due to a disaster or the like. 

[138] After having received the instruction issued from the sub-host computer 1020, the 
sub-DKC 1040 judges whether or not the sub-freezing mode 1090 is 1 (Step 3010). If it is 
15 determined in Step 1040 that the sub-freezing mode 1090 is 1, then the sub-DKC 1040 

suspends the sub-freezing processing 2800 which the sub-DKC 1040 itself is executing (Step 
3040) to set to zero the sub-freezing mode (Step 3050). 

[139] The sub-DKC 1040 initializes all of the values of the sub-differential bitmap to zero 
(Step 3060) to erase the information which the sub-differential volume A 1 170 and the sub- 
20 differential volume B 1 175 have (Step 3070). 

[140] The sub-DKC 1040 transmits the completion report to the sub-host computer 1020 
(Step 3080) to complete the sub-recovery processing 3000. 

[141] On the other hand, if it is determined in Step 3010 that the sub-freezing mode 1090 is 
not 1, then the sub-DKC 1040 judges whether or not the sub-freezing mode 1090 is zero 
25 (Step 3020). If it is determined in Step 3020 that the sub-freezing mode 1090 is zero, then 
the sub-DKC 1040 executes the processing in Step 3060. On the other hand, if it is 
determined in Step 3020 that the sub-freezing mode 1090 is not zero, then the sub-DKC 1040 
waits for the sub-freezing mode 1090 to become zero to execute the processing in Step 3020 
(Step 3030). 

30 [142] By executing the sub-recovery processing 3000, the volume image of the main 

volume 1 120 at a time point when the freeze instruction was issued from the host computer 
1010 to the main DKC 1030 last time or this time is copied to the sub-volume 1 150. Then, 
the sub-host computer 1020 can use freely the copied volume image. 

22 



[143] Fig. 10 is a flow chart useful in explaining a read processing 3200 which the main 
DKC 1030 executes at the time when the host computer 1010 has issued the read command to 
read the data to the main storage apparatus system 1 1 80. 

[144] After having received the read command issued from the host computer 1010, the 
5 main DKC 1030 judges whether or not the data which has been requested from the host 
computer 1010 is present in the RAM 1260 (Step 3205). 

[145] If it is determined in Step 3205 that the requested data is not present in the RAM 
1260, then the main DKC 1030 checks the main freezing mode (Step 3210). If it is 
determined in Step 3210 that the main freezing mode 1060 is OFF, then the main DKC 1030 

10 reads out the requested data from the main volume 1 130 (Step 3220) to transfer the data of 
interest to the host computer 1010 to complete the read processing 3200 (Step 3230). 
[146] On the other hand, if it is determined in Step 3210 that the main freezing mode 1060 
is ON, then the main DKC 1030 judges whether or not the bit of the main differential bitmap 
1080 corresponding to the data requested from the host computer 1010 is 1 (Step 3240). 

1 5 [147] If it is determined in Step 3240 that the corresponding bit is zero, then the main DKC 
1030 executes the processing in Step 3220. On the other hand, if it is determined in Step 
3240 that the corresponding bit is 1, then the main DKC 1030 finds out the data, which has 
been required from the host computer 1010, from the main differential volume 1 140 to read 
out the data thus found out to execute the processing in Step 3230 (Step 3250). 

20 [148] On the other hand, if it is determined in Step 3205 that the requested data is present in 
the RAM 1260, then the main DKC 1030 executes the processing in Step 3230. 
[149] Fig. 1 1 is a schematic view useful in explaining the transmission and the reception of 
the freeze instruction between the apparatuses included in the host computer 1 000. 
[150] In Fig. 1 1 , the vertical axis represents the time base. Thus, it is meant that the time 

25 elapses as the position on the drawing is located more downwardly. 

[151] The host computer 1010 issues the freeze instruction to the main DKC 1030, and then 
in response to the freeze instruction issued thereto, the main DKC 1030 issues the freeze 
instruction to the sub-DKC 1040. 

[152] After having transferred the volume image of the main volume 1 120 at a time point of 
30 the reception of the freeze instruction to the sub-DKC 1040, the main DKC 1030 transmits 
the report showing the completion of the transfer of all of the freezing data to the sub-DKC 
1040. 

[153] The sub-DKC 1040 reflects the data of the main volume 1 120 at a time point of the 
issue of the freeze instruction to the sub-volume 1 150, and after completion of the reflection 
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of the data, informs the main DKC 1030 of the completion of the freezing. In response 
thereto, the main DKC 1030 informs the host computer 1010 of the completion of the 
freezing. 

[154] By adopting the configuration of the present embodiment, the data which is held in 
5 the main disk unit at a time point when the freeze instruction is issued from the host computer 
1010 can be held in the sub-disk unit, and hence even when the data is destroyed, the loss of 
the data can be suppressed to a minimum. 

[155] In other words, the sub-storage apparatus system 1 190 holds the data at a time point 
of the issue of the freeze instruction and at the same time, holds as the differential data the 
1 0 data which was written on and after that time point. This freezing is carried out at periodic 
intervals, whereby when the data of the main storage apparatus system 1 180 is not able to be 
used due to for some cause or other, all of the data at a time point when it was frozen last 
time can be obtained from the sub-storage apparatus system 1 190. 

[156] In this connection, while in the present embodiment, the description has been given 
1 5 on the assumption that a plurality of volumes are present inside of the main disk unit 1 130, 
the present invention is not intended to be limited thereto. That is, a plurality of disk units 
may be present in the main storage apparatus system 1 1 80, and each of the disk units may 
take the function of the volume on itself. This is also applied to the sub-disk unit 1 160. 
[157] Fig. 12 is a block diagram showing a configuration of a second embodiment of the 
20 computer system 1000 to which the present invention is applied. 

[158] A point of difference of the present embodiment from the first embodiment is that the 
copy of the coherent volume image at the time when the freeze instruction has been issued 
from the host computer 1010 is generated in the main disk unit 1 130 once, and the data is 
transferred from the volume thus generated to the sub-DKC. 
25 [159] The present embodiment is also different from the first embodiment in that two 
volumes are prepared for the sub-storage apparatus system, and one of them is made the 
volume which is used in order to generate the coherent volume. 

[160] The description will hereinafter be given with respect to points of difference from the 
computer system 1000 of the first embodiment. The parts which will not be described here 
30 have the same configurations as those of the first embodiment, and hence are designated with 
the same reference numerals. 

[161] A main disk unit 1 130' includes a main main-volume 1300 and a main sub-volume 
1310. 
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[162 J In the main main- volume 1300, there is stored the data which has been transferred 
from the host computer 1010. The main sub-volume 1310 is used when copying the volume 
image of the main main- volume 1300 at a time point when the freeze instruction was issued. 
The blocks which the main main- volume 1300 and the main sub-volume 1310 have, 
5 respectively, show one-to-one correspondence. 

[163] The main DKC 1030' has a main bitmap A 1340, a main bitmap B 1350, and a main 
bitmap C 1360 all of which are contained in the RAM 1260. 

[164] The main bitmap A 1340, when the contained bit is 1, with respect to the block 
corresponding to that bit, shows that the contents of the data which is stored in the main 
10 main-volume 1300 are different from those of the data which is stored in the main sub- 
volume 1310. 

[165] While the main bitmap B 1350 also shows the difference between the main main- 
volume 1300 and the main sub-volume 1310, only the bit corresponding to the block in which 
after the freeze instruction has been transmitted from the host computer 1010, the data is 
15 written to the main main- volume 1300 becomes 1. 

[166] The main bitmap C 1360 shows that there is the difference in the contents between the 
main sub-volume 1310 and a sub-main- volume 1320. 

[167] A sub-disk unit 1 160' includes the sub-main- volume 1320 and a sub-sub-volume 
1330. 

20 [168] To the sub-main- volume 1320, there is written the data which has been transferred 
from the main DKC 1030'. In the sub-sub-volume 1330, there is built the volume image of 
the main main- volume 1300 at a time point when the freezing processing was executed last 
time. 

[169] A sub-DKC 1040' has a sub-bitmap A 1370 showing the difference between the sub- 
25 main-volume 1320 and the sub-sub-volume 1330. Then, the sub-bitmap A 1370 is contained 
in the RAM 1260. In the present embodiment, the sub-freezing mode 1090 is not used. 
[170] Figs. 13A to 13D are respectively schematic views showing the flow of the data 
corresponding to the present embodiment. 

[171] In Fig. 13 A, there is shown the flow of the data in the case where the host computer 
30 1010 does not issue the freeze instruction. 

[172] The data which has been transferred from the host computer 1010 is written to the 
main main- volume 1300 (indicated by an arrow J 190). 
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[173] Then, it is assumed that in the sub-sub-volume 1330, there is stored the data of the 
volume image of the main main- volume 1300 at a time point when the host computer 1010 
issued the freeze instruction last time. 

[174] In Fig. 13B, there is shown the flow of the data for a time period ranging from a time 
5 point when a host computer 1010 has issued the freeze instruction up to a time point when the 
main storage apparatus system 1 180 generates, in the main sub-volume 1310, the copy of the 
volume image of the main main- volume 1300 at a time point when the main storage 
apparatus system 1 180 issued the freeze instruction. 

[175] The main disk unit 1 130 copies the data from the main main-volume 1300 to the main 

1 0 sub- volume 1310 (indicated by an arrow L 2 1 0). 

[176] While the data which the main DKC 1030' has received is written to the main main- 
volume 1300 (indicated by an arrow K 200), in the case of writing of the data to the block in 
which the copy to the main sub-volume 1310 is not yet carried out, after completion of the 
processing of copying the data indicated by the arrow L 210, the data is written to the main 

15 main- volume 1300. 

[177] In Fig. 13C, there is shown the flow of the data for a time period ranging from a time 
point when the volume image of the main main- volume 1300 at a point when the host 
computer 1010 issued the freeze instruction has been generated in the main sub- volume 1310 
up to a time point when the volume image generated in the main sub-volume 1310 is 

20 generated in the sub-main- volume 1320 which the sub-DKC 1040' manages. 

[178] The data which has been transferred from the host computer 1010 is written to the 
main main- volume 1300 (indicated by the arrow M 220). The data of the main sub-volume 
1310 is read out to the main DKC 1030' to be transferred to the sub-DKC 1040' to be written 
to the sub-main- volume 1320 (indicated by the arrow N 230). 

25 [179] In Fig. 13D, there is shown the flow of the data for a time period ranging from a time 
point when the volume image of the main main-volume 1300 at a time point when the host 
computer 1010 issued the freeze instruction to the sub-main- volume 1320 has been generated 
up to a time point when the volume image thus generated is copied to the sub-sub-volume 
1330. 

30 [180] The data which has been transferred from the host computer 1010 is written to the 
main main- volume 1300 (indicated by an arrow O 240). 

[181] The sub-DKC 1040' controls the copy of the data from the sub-main-volume 1320 to 
the sub-sub-volume 1330 (indicated by an arrow P 250). 
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[182] In the present embodiment, the copy of the data for duplicating the volume image of 
the main main-volume 1300 is carried out in such a way that of the data which is stored in the 
source volume and in the destination volume, only the differential data is subjected to the 
differential copy. 

5 [183] In Figs. 13 A, 13B and 13C, the sub-sub-volume 1330 holds the volume image of the 
main main- volume 1300 at a time point when the host computer 1010 issued the freeze 
instruction last time. 

[184] In Fig. 13D, the sub-main-volume 1320 holds the volume image of the main main- 
volume 1300 at a time point when the host computer 1010 has issued the freeze instruction 
10 this time. 

[185] Therefore, volume image of the main main-volume 1300 at a time point when the host 
computer 1010 issued the freeze instruction will be held in any case. 

[186] Fig. 14 is a schematic view useful in explaining the flow of the freeze instruction in 
the computer system 1000 of the present embodiment. In the figure, the vertical direction 
1 5 represents the time base. Thus, it is meant that the time elapses as the position on the 
drawing is located more downwardly. 

[187] The host computer 1010 issues the freeze instruction to the main DKC 1030'. In 
response to the freeze instruction issued thereto, the main DKC 1030' differential-copies 
(hereinafter, the differential copy will be referred to as "the split", when applicable) all of the 

20 data, which is stored in the main main- volume 1300 at a time point when the freeze 

instruction was issued, to the main sub-volume 1310. After completion of the differential 
copy, the main DKC 1030' reads out the data of the main sub-volume 1310 to transmit the 
data thus read out to the sub-DKC 1040'. Then, the sub-DKC 1040' writes the data which 
has been received from the main DKC 1030' to the sub-main volume 1320. 

25 [188] The main DKC 1030' issues the freeze instruction to the sub-DKC 1040'. After 
having received the freeze instruction, the sub-DKC 1040' splits the volume image of the 
sub-main volume 1320 to the sub-sub-volume 1330. After completion of the split, the sub- 
DKC 1040' transmits the report exhibiting the completion of the freezing to the main DKC 
1030'. After having received this report, the main DKC 1030' informs the host computer 

30 1010 of the completion of the freezing. 

[189] Fig. 15 is a flow chart useful in explaining a write processing A 3400 which the main 
DKC 1030' executes when the host computer 1010 transmits the data to the main storage 
apparatus system 1180. 
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[190] The main DKC 1030' receives the write data from the host computer 1010 (Step 
3410) to judge whether or not the main freezing mode 1060 is in the ON state (Step 3420). 
[191] If it is determined in Step 3420 that the main freezing mode 1060 is in the ON state, 
then the main DKC 1030' judges whether or not the bit of the main bitmap A 1340 
5 corresponding to the block in which the transferred data is stored is 1 (Step 3430). 

[192] If it is determined in Step 3420 that the bit of the main bitmap A 1340 is zero, then the 
main DKC 1030' executes the processing in Step 3470. On the other hand, if it is determined 
in Step 3420 that the bit of the main bitmap A 1340 is 1, then the main DKC 1030' controls 
the main disk unit 1 130 in such a way as to copy the data stored in the block corresponding to 
10 the bit of 1 from the main main volume 1300 to the main sub-volume 1310 (Step 3440). 

[193] After completion of the copy, the main DKC 1030' sets to zero the corresponding bit 
of the main bitmap A 1340 (Step 3450) and also sets to 1 the corresponding bit of the main 
bitmap C 1360 (Step 3460). 

[194] The main DKC 1030' sets to 1 the bit of the main bitmap B 1350 corresponding to the 
15 block in which the transferred data is stored (Step 3470) to write the write data to the 

corresponding block of the main main- volume 1300 to complete the write processing A 3400 
(Step 3480). 

[195] On the other hand, if it is determined in Step 3420 that the main freezing mode is in 
the OFF state, then the main DKC 1030' sets to 1 the corresponding bit of the main bitmap A 
20 to execute the processing in Step 3480 (Step 3490). 

[196] Fig. 16 is a flow chart useful in explaining the main freezing processing A 3600 
which the main DKC 1030' executes when the host computer 1010 has issued the freeze 
instruction to the main DKC 1030'. 

[197] After having received the freeze instruction from the host computer 1010 (Step 3610), 
25 the main DKC 1030' makes the ON state the main freezing mode (Step 3620) to judge 
whether or not all of the bits of the main bitmap A 1 1340 is zero (Step 3630). 
[198] If it is determined in Step 3630 that all of the bits of the main bitmap A 1340 are not 
zero, then the main DKC 1030' specifies the block in which the bit is 1 in the main bitmap A 
1340 (Step 3640). Then, the main DKC 1030' reads out the specified block of the main main 
30 volume 1300 to control the main disk 1 130' in such a way as to copy the block thus read out 
to the main sub-volume 1310 (Step 3650). 

[199] After completion of the copy of the specified block, the main DKC 1030' sets to zero 
the corresponding bit of the main bitmap A 1340 (Step 3660), while sets to 1 the 
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corresponding bit of the main bitmap C 1360 to return back to the processing in Step 3630 
(Step 3670). 

[200] If it is determined in Step 3630 that all of the bits of the main bitmap A 1340 are zero, 
then the main DKC 1030' judges whether or not all of the bits of the bitmap C1360 are zero 
5 (Step 3680). 

[201] If it is determined in Step 3680 that all of the bits of the main bitmap CI 360 are not 
zero, then the main DKC 1030' selects the bit as 1 of the main bitmap C 1360 (Step 3690) to 
read out the block corresponding to the bit of the main sub-volume 13 10 to transmit the block 
thus read out to the sub-DKC 1040' (Step 3700). 
10 [202] The main DKC 1030' sets to zero the corresponding bit of the main bitmap C 1360 to 
return back to the processing in Step 3680 (Step 3710). 

[203] On the other hand, if it is determined in Step 3680 that all of the bits of the main 
bitmap C 1360 are zero, then the main DKC 1030' issues the freeze instruction to the sub- 
DKC 1040' (Step 3720) to wait for the report exhibiting the freezing completion to be 

15 transmitted thereto from the sub-DKC 1040' (Step 3730). 

[204] After having received the report exhibiting the completion of the freezing from the 
sub-DKC 1040' (Step 3740), the main DKC 1030' copies the contents of the main bitmap B 
1350 to the main bitmap A 1340 (Step 3750) to set to zero all of the bits of the main bitmap B 
1350 (Step 3760). The main DKC 1030' makes the OFF state the main freezing mode 1060 

20 (Step 3770) to transmit the report exhibiting the freezing completion to the host computer 
1010 to complete the main freezing processing A 3600 (Step 3780). 
[205] Fig. 1 7 is a flow chart useful in explaining the sub-remote copy processing A 3800 
which the sub-DKC 1040 executes when the processing of the remote copy is executed for 
the sub-DKC 1040 in Step 3700 of the main freezing processing A 3600. 

25 [206] After having received the data from the main DKC 1030' (Step 3810), the sub-DKC 
1040 controls the sub-disk unit 1 160 in such a way that the received data is written to the 
corresponding block of the sub-main-volume 1320 (Step 3820). 

[207] After the processing of writing the data has been completed, the sub-DKC 1040 sets 
to 1 the corresponding bit of the sub-bitmap A 1370 to complete the sub-remote copy 
30 processing A 3800 (Step 3830). 

[208] Fig. 18 is a flow chart useful in explaining a sub-freezing processing A 4000 which 
the sub-DKC 1040' executes when the main DKC 1030' has issued the freeze instruction to 
the sub-DKC 1040'. 
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[209] The sub-DKC 1040' which has received the freeze instruction from the main DKC 
1030' judges whether or not all of the bits of the sub-bitmap A 1370 are zero (Step 4010). 
[210] If it is determined in Step 4010 that all of the bits of the sub-bitmap A 1370 are not 
zero, then the sub-DKC 1040' controls the sub-disk unit 1 160' in such a way that it selects 
5 the block of the sub-main-volume 1320 corresponding to the bit as 1 of the sub-bitmap A 
1370 (Step 4020) to read out the block thus selected to copy the block thus read out to the 
corresponding block of the sub-sub-volume 1330 (Step 4030). 

[211] After completion of the copy of the corresponding block, the sub-DKC 1040' sets to 
zero the corresponding bit of the sub-bitmap A 1370 to return back to the processing in Step 
10 4010 (Step 4040). 

[212] On the other hand, if it is determined in Step 4010 that all of the bits of the sub- 
bitmap A 1370 are zero, then the sub-DKC 1040' transmits the report exhibiting the 
completion of the freezing to the main DKC 1030' to complete the sub-freezing processing A 
4000 (Step 4050). 

15 [213] In the present embodiment, since the data of the volume image of the main main- 
volume 1300 at a time point when the host computer 1010 issued the freeze instruction last 
time is stored in either the sub-sub-volume 1330 or the sub-main-volume 1320, the 
processing corresponding to the sub-recovery processing 3000 become unnecessary. 
[214] When the read request has been made from the host computer 1010, the processing(s) 

20 as in the first embodiment does(do) not need to be executed, and the data of the main main- 
volume 1300 has only to be transferred. 

[215] Fig. 19 is a block diagram showing a configuration of a third embodiment of the 
computer system 1000 to which the present invention is applied. 

[216] In the present embodiment, a point that the remote copy is carried out only in Fig. 
25 13C of Figs. 13A to 13D and hence the network can not be used effectively in the second 
embodiment is improved. 

[217] The description will hereinafter be given with respect to only a point of difference 
from the second embodiment. 

[218] A main DKC 1030" has a main bitmap D 1380 which the RAM 1260 includes. 
30 [219] The main bitmap D 1380 shows the data which is not yet copied to the main sub- 
volume 1310 and which is stored in the main main- volume 1300, i.e., the block containing 
that data which was remote-copied to the sub-main-volume 1320 before issue of the freeze 
instruction. 

[220] A sub-DKC 1040" has a sub-bitmap B 1390. 

30 



[221] The sub-bitmap B 1390 shows the block which is remote-copied from a main DKC 
1030" to the sub-main volume 1320 on and after the freeze instruction has been issued this 
time but before the freeze instruction will be issued next time. 

[222] In the present embodiment, the main freezing mode takes the integral number from 0 
5 to 3. The sub-freezing mode shows either ON or OFF. 

[223] Figs. 20A to 20D are respectively schematic views showing the flow of the data in the 
computer system 1000 of the present embodiment. 

[224] The description will hereinafter be given with respect to the present embodiment 
while comparing the present embodiment with Figs. 13A to 13D showing the flow of the data 

10 in the computer system 1000 to which the second embodiment is applied. 

[225] In Fig. 2 OA, the data which has been written from the host computer 1010 to the main 
main- volume 1300 is transferred to the sub-DKC 1040" before the host computer 1010 issues 
the freeze instruction (indicated by an arrow R 270). An arrow Q 260 corresponds to the 
arrow J 190 shown in Fig. 13 A. 

1 5 [226] An arrow S 280 and an arrow T 290 in Fig. 20B show the flow of the data 
corresponding to the arrow K 200 and the arrow L 210 in Fig. 13B, respectively. 
[227] The data which has already been copied (indicated by the arrow T 290) to the main 
sub-volume 1310 but which is not yet sent to the sub-main-volume 1320 is transferred from 
the corresponding block of the main sub-volume 1310 to the sub-DKC 1040" through the 

20 main DKC 1030" to be written to the sub-main- volume 1320 (indicated by an arrow K 300). 
[228] An arrow V 3 10 and an arrow W 320 shown in Fig. 20C correspond to the arrow M 
220 and the arrow N 230 shown in Fig. 13C, respectively. 

[229] An arrow X 330 and an arrow Z 350 shown in Fig. 20D correspond to the arrow O 
and the arrow P shown in Fig. 13, respectively. 
25 [230] The data which has been newly written to the main main- volume 1300 (indicated by 
the arrow X 330) is copied to the sub-main- volume 1320 (indicated by an arrow Y 340) 
before the host computer 1010 issues the next freeze instruction. 

[231] In the case where the data for the block which is not yet copied from the sub-main- 
volume 1320 to the sub-sub-volume 1300 has been received from the main DKC 1030" by 
30 the sub-DKC 1040", the sub-DKC 1040" copies the data in the block corresponding to the 
data which has been received once from the sub-main-volume 1320 to the sub-sub-volume 
1330 (indicated by the arrow Z 350). Thereafter, the sub-DKC 1040" controls the sub-disk 
unit 1 160' in such a way that the data is written to the corresponding block of the sub-main- 
volume 1320. 



[232] Fig. 21 is a schematic view useful in explaining the flow of the freeze instruction in 
the computer system 1000 to which the present embodiment is applied. In the figure, the 
vertical direction represents the time base, and hence the time elapses as the position on the 
drawing is located more downwardly. 
5 [233] Comparing Fig. 21 with Fig. 14, it is understood that a point of difference is that the 
remote copy is carried out for a longer time period. 

[234] Fig. 22 is a flow chart useful in explaining a write processing B 4200 which the main 
DKC 1030" executes when the data has been transferred from the host computer 1010 to the 
main storage apparatus system 1 1 80. 

10 [235] The main DKC 1030" receives the write data from the host computer 1010 (Step 
4210) to judge whether or not the main freezing mode 1060 is zero (Step 4220). If it is 
determined in Step 4220 that the main freezing mode 1060 is not zero, then the main DKC 
1030" judges whether or not the main freezing mode 1060 is 1 (Step 4230). 
[236] If it is determined in Step 4230 that the main freezing mode 1060 is not 1, then the 

15 main DKC 1030" judges whether or not the main freezing mode 1060 is 2 (Step 4240). 

[237] If it is determined in Step 4240 that the main freezing mode 1060 is 2, then the main 
DKC 1030" executes the processing in Step 4260. On the other hand, if it is determined in 
Step 4240 that the freezing mode 1060 is not 2 (i.e., it is determined to be 3), then the main 
DKC 1030" sets to zero the bit of the main bitmap D 1380 corresponding to the transferred 

20 block (Step 4250). 

[238] The main DKC 1030" sets to 1 the bit corresponding to the write data of the main 
bitmap A 1340 (Step 4260) and controls the main disk unit 1 130" in such a way that the write 
data is stored in the corresponding block of the main main- volume 1300 to complete the main 
write processing B 4200 (Step 4270). 

25 [239] On the other hand, if it is determined in Step 4220 that the main freezing mode 1060 
is zero, then the main DKC 1030" sets to zero the corresponding bit of the main bitmap C 
1380 to execute the processing in Step 4250 (Step 4280). 

[240] In addition, if it is determined in Step 4230 that the main freezing mode 1060 is 1, 
then the main DKC 1030" judges whether or not the corresponding bit of the main bitmap A 
30 1340 is 1 (Step 4290). If it is determined in Step 4290 that the corresponding bit is zero, then 
the main DKC 1030" executes the processing in Step 4330. On the other hand, if it is 
determined in Step 4290 that the corresponding bit is 1, then the main DKC 1030" controls 
the main disk unit 1 160' in such a way that the data of the block of the main main-volume 
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1300 corresponding to the bit of interest is copied to the corresponding block of the main sub- 
volume 1310 (Step 4300). 

[241] The main DKC 1030" sets to zero the bit of the main bitmap A 1340 corresponding to 
the copied data (Step 4310). The main DKC 1030" sets to 1 the bit of the main bitmap C 
5 1360 corresponding to the copied data (Step 4320). In addition, the main DKC 1030" sets to 
1 the bit of the main bitmap B 1350 corresponding to the copied data (Step 4330) to return 
back to the processing in Step 4270. 

[242] Fig. 23 is a flow chart useful in explaining a main freezing processing B 4400 which 
the main DKC 1030" executes when the host computer 1010 has issued the freeze instruction 
10 to the main DKC 1030". 

[243] After having received the freeze instruction from the host computer 1010 (Step 4410), 
the main DKC 1030" sets to 1 the main freezing mode 1060 to complete the main freezing 
processing B 4400 (Step 4420). 

[244] Fig. 24 is a flow chart useful in explaining a main copy processing 4600 which is 
1 5 activated by the main DKC 1 030" at the time when turning ON the power source of the main 
storage apparatus system 1 180, and thereafter the main DKC 1030" continues to execute as 
one task. 

[245] The main DKC 1030" judges whether or not the main freezing mode 1060 is 1 (Step 
4610). If it is determined in Step 4610 that the main freezing mode 1060 is 1, then the main 
20 DKC 1030" judges whether or not the bit of 1 is present in the bits contained in the main 
bitmap A 1340 (Step 4620). 

[246] If it is determined in Step 4610 that the bit of 1 is present in the bits contained in the 
main bitmap A 1340, then the main DKC 1030" controls the main disk unit 1 130" in such a 
way that the block corresponding to the bit of interest is selected (Step 4630), and the data of 
25 the block corresponding to the bit of the main main volume 1300 thus selected is copied to 
the corresponding block of the main sub-volume 1310 (Step 4640). 

[247] After completion of the copy, the main DKC 1030" sets to zero the corresponding bit 
of the main bitmap A 1340 (Step 4650), while sets to 1 the corresponding bit of the main 
bitmap C 1360 to return back to the processing in Step 4620 (Step 4660). 
30 [248] On the other hand, if it is determined in Step 4610 that the main freezing mode is not 
1, then the main DKC 1030" executes the processing in Step 4680. 
[249] In addition, if it is determined in Step 4620 that the bit of 1 is absent in the main 
bitmap A 1340, then the main DKC 1030" sets to 2 the main freezing mode 1060 (Step 
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4670), and carries out waiting for some time to return back to the processing in Step 4610 
(Step 4680). 

[250] Fig. 25 is a flow chart useful in explaining a main remote copy processing 4800 which 
is activated by the main DKC 1030" at the time when turning ON the power source of the 

5 main source apparatus system and which the main DKC 1030" executes as one task. 

[251J The main DKC 1030" judges whether the main freezing mode 1060 is zero or 3 (Step 
4810). If it is determined in Step 4810 that the main freezing mode 1060 is zero or 3, then 
the main DKC 1030" judges whether or not the block in which the bit of the main bitmap A 
1340 is 1 is present (Step 4820). If so, the main DKC 1030" selects the block of interest 

10 (Step 4830). 

[252] The main DKC 1 030" sets to 1 the bit of the main bitmap D 1 3 80 corresponding to 
the selected block (Step 4840) to read out the corresponding block from the main main- 
volume 1300 to transmit the block thus read out to the sub-DKC 1040" (Step 4850). 
[253] The main DKC 1030" receives the report exhibiting the reception of the data from the 
1 5 sub-DKC 1 040" to return back to the processing in Step 48 1 0 (Step 4860). 

[254] On the other hand, if it is determined in Step 4820 that the block in which the bit of 
the main bitmap A 1340 is 1 is absent, then the main DKC 1030" carries out step of waiting 
for some time (for about several milliseconds) to return back to the processing in Step 4810 
(Step 4870). 

20 [255] In addition, if it is determined in Step 48 10 that the main freezing mode 1060 is 

neither zero nor 3, then the main DKC 1030" judges whether or not the block is present in 
which the main bitmap C 1360 is 1 and also the main bitmap D 1380 is zero (Step 4880). 
[256] If it is determined in Step 4880 that the block fulfilling the above-mentioned 
conditions is present, then the main DKC 1030" selects the block of interest (Step 4890) to 

25 set to zero the corresponding bit of the main bitmap C 1360 (Step 4900). The main DKC 

1030" reads out the block thus selected from the main sub-volume 1310 to transfer the block 
thus read out to the sub-DKC 1040" (Step 4910). 

[257] The main DKC 1030" receives the report exhibiting the reception of the data from the 
sub-DKC 1040" to return back to the processing in Step 4810 (Step 4920). 
30 [258] On the other hand, if it is determined in Step 4880 that the block is not present in 
which the main bitmap C 1360 is 1 and also the main bitmap D 1380 is zero, then the main 
DKC 1030" judges whether or not the main freezing mode 1060 is 2 (Step 4930). 
[259] If it is determined in Step 4930 that the main freezing mode 1060 is 2, then the main 
DKC 1030" clears all of the bits in the main bitmap D 1380 to zero (Step 4940), and sets to 3 
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the main freezing mode 1060 (Step 4950) and then issues the freeze instruction to the sub- 
DKC 1040" to return back to the processing in Step 4810 (Step 4960). 
[260] On the other hand, if it is determined in Step 4930 that the main freezing mode 1060 
is not 2, then the main DKC 1030" carried out the step of waiting for some time (for about 
5 several milliseconds) to return back to the processing in Step 4810 (Step 4970). 

[261] Fig. 26 is a flow chart useful in explaining a main freezing completion processing 
5000 which the main DKC 1030" executes at the time when having received the report 
exhibiting the completion of the freezing from the sub-DKC 1040". 
[262] After having received the report exhibiting the freezing completion from the sub- 
10 DKC 1040" (Step 5010), the main DKC 1030" sets to zero the main freezing mode 1060 
(Step 5020) and informs the host computer 1010 of the freezing completion to complete the 
main freezing completion processing 5000 (Step 5030). 

[263] Fig. 27 is a flow chart useful in explaining a sub-freezing processing 5200 which the 
sub-DKC 1040" executes at the time when the main DKC 1030" has issued the freeze 

1 5 instruction to the sub-DKC 1040". 

[264] After having received the freeze instruction from the main DKC 1030" (Step 5210), 
the sub-DKC 1040" makes the ON state the sub-freezing mode 1090 (Step 5220) to judge 
whether or not the bit of 1 is present in the sub-bitmap A 1370 (Step 5230). 
[265] If it is determined in Step 5230 that the bit of 1 is present in the sub-bitmap A 1370, 

20 then the sub-DKC 1040" controls the sub-disk unit 1 160 5 in such a way that the block 

corresponding to the bit of interest is selected (Step 5240), and the block thus selected of the 
sub-main- volume 1320 is read out to be copied to the corresponding block of the sub-sub- 
volume 1330 (Step 5250). 

[266] The sub-DKC 1040" sets to zero the corresponding bit of the sub-bitmap A 1370 to 
25 return back to processing in Step 5230 (Step 5260). 

[267] On the other hand, if it is determined in Step 5230 that the bit of 1 is absent in the 
sub-bitmap A 1370, then the sub-DKC 1040" copies the sub-bitmap B 1390 to the sub- 
bitmap A 1370 (Step 5270) to make all of the bits of the sub-bitmap B 1390 to zero (Step 
5280). 

30 [268] The sub-DKC 1040" makes the OFF state the sub-freezing mode 1090 (Step 5290) 
and informs the main DKC 1040" of the freezing completion to complete the sub-freezing 
processing 5200 (Step 5300). 
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[269] Fig. 28 is a flow chart useful in explaining a sub-remote copy processing B 5400 
which the sub-DKC 1040" executes at the time when in Step 4910 in the main remote copy 
processing B 4800, the main DKC 1030" remote-copies the data to the sub-DKC 1040". 
[270] After having received the data from the main DKC 1030" (Step 5410), the sub-DKC 
5 1 040" informs the main DKC 1 030" of the reception of the data (Step 5420). 

[271] Thereafter, the sub-DKC 1040" judges whether or not the sub-freezing mode 1090 is 
in the ON state (Step 5430). If it is determined in Step 5430 that the sub-freezing mode 1090 
is in the ON state, then the sub-DKC 1040" judges whether or not that the bit of 1 is present 
in the sub-bitmap A 1370 (Step 5440). If it is determined in Step 5440 that the bit of 1 is 
10 absent in the sub-bitmap A 1370, then the sub-DKC 1040" executes the processing in Step 
5470. 

[272] On the other hand, if it is determined in Step 5440 that the bit of 1 is present in the 
sub-bitmap A 1370, then the sub-DKC 1040" controls the sub-disk unit 1 160" in such a way 
that the block of the sub-main- volume 1320 corresponding to the bit of 1 is read out to be 

15 copied to the corresponding block of the sub-sub-volume 1330 (Step 5450). 

[273] The sub-DKC 1040" sets to zero the corresponding bit of the sub-bitmap A 1370 
(Step 5460), while sets to 1 the corresponding bit of the sub-bitmap B 1390 (Step 5470). The 
sub-DKC 1040" controls the sub-disk unit 1 160' in such a way that the received data is 
written to the sub-main volume 1320 to complete the sub-remote copy processing B 5400 

20 (Step 5480). 

[274] On the other hand, if it is determined in Step 5430 that the sub-freezing mode 1090 is 
in the OFF state, then the sub-DKC 1040" makes 1 the corresponding bit of the sub-bitmap A 
1370 to execute the processing in Step 5480 (Step 5490). 

[275] In the present embodiment, since the volume image of the main main-volume 1300 at 
25 a time point when the host computer 1010 issued the freeze instruction last time is held in the 
form of the data stored in the sub-sub-volume 1330 or in the form of the combination of the 
data which is stored in the sub-sub-volume 1330 and the sub-main-volume 1320, the 
processing corresponding to the sub-recovery processing 3000 becomes unnecessary. 
[276] When the request to read the data has been made from the host computer 1010, the 
30 data of the main main-volume 1 300 may be transferred thereto. 

[277] Next, a fourth embodiment of the present invention will hereinafter be described. 
[278] A point of difference of the fourth embodiment from other embodiments is that the 
data which is transferred from the main storage apparatus system 1 130 to the sub- storage 
apparatus system 1 160 is encrypted. 
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[279] The fourth embodiment will now be described on the basis of the second embodiment. 
But, it goes without saying that in the first embodiment and the third embodiment as well, the 
present embodiment can be adopted. 

[280] In the present embodiment, when the data is transferred from the main main-volume 
5 1300 to the main sub-volume 1310, the data to be transferred is encrypted (encoded). 

[281] In addition, the encrypted data is transferred from the main sub-volume 1310 to the 
sub-main- volume 1320, and when the encrypted data is copied from the sub-main- volume 
1320 to the sub-sub- volume 1330, the data of interest is decoded. 
[282] As a result, the encrypted data will be transferred on the network 1050. The 
10 description will hereinbelow be given with respect to a point of difference from the second 
embodiment. 

[283] In the present embodiment, in Fig. 13B, the data is copied from the main main- 
volume 1300 to the main sub-volume 1310 while the data is encrypted (encoded) (indicated 
by an arrow L 210). 

15 [284] In addition, in Fig. 13D, in the sub-DKC 1040', the data is copied from the sub-main- 
volume 1320 to the sub-sub-volume 1330 while the encrypted data is decoded (indicated by 
an arrow P 250). 

[285] In Fig. 14, the stored data is encrypted when the main DKC 1030' is split. In 
addition, when the sub-DKC 1040' is split, the stored data which is already encrypted is 
20 decoded. 

[286] In Step 3440 shown in Fig. 15, the main DKC 1030' copies the data stored in the 
corresponding block from the main main-volume 1300 to the main sub-volume 1310 after the 
encryption of the data of interest. 

[287] In Step 3650 shown in Fig. 16, the main DKC 1030' controls the main disk unit 1 130' 
25 in such a way that the corresponding block of the main main-volume 1300 is read out to be 
encrypted, and the encrypted data is copied to the main sub- volume 1310. 
[288] In Step 4030 shown in Fig. 18, the sub-DKC 1040' controls the sub-disk unit 1 160' in 
such a way that the corresponding block of the sub-main- volume 1320 is read out to the sub- 
DKC 1040' and the encrypted data is decoded to be copied to the corresponding block of the 
30 sub-sub-volume 1330. 

[289] By adopting the configuration of the present embodiment, the volume image in the 
freezing can be ensured in any one of the disk units while ensuring the safety of the data 
which is being transferred. 
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[290] By adopting the configuration of the present invention, even when the remote copy of 
no guarantee to order is carried out, the volume image having excellent coherency can be 
ensured. 

[291] While the present invention has been particularly shown and described with reference 
to the preferred embodiments, it will be understood that the various changes and 
modifications will occur to those skilled in the art without departing from the scope and true 
spirit of the invention. The scope of the invention is therefor to be determined solely by the 
appended claims. 
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